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What  is  “Acquisition” 


Operational 
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A  Strategic  Partnership 
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The  State  of  Acquisition  Practice 


The  agencies  assume  the  partnership  arrangement  absolves  them  of  all 
acquisition  management  responsibilities... 

Virtually  all  (Air  Force)  software-intensive  systems  suffer  from 
difficulties  achieving  cost,  schedule,  and  performance  objectives,  gao 

“I'd  rather  have  it  wrong  than  have  it  late.”  A  senior  manager  (industry) 

“The  bottom  line  is  schedule.  My  promotions  and  raises  are  based  on 

meeting  schedule  first  and  foremost.”  A  program  manager  (government) 

Lack  of  robust  systems  engineering  practices  identified  as  critical 
factor  in  SBIRS-High  problems. 
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Example  Program 

Background 

Large  DoD  program  with  multiple,  geographically  dispersed  engineering 
locations. 

Multi-contractor  teams  (10+)  using  different  processes. 

Several  million  lines  of  code. 

Systems  engineering  challenges. 

Combination  of  legacy,  re-use,  COTS  integration  and  new  development. 

All  contractor  sites  are  Maturity  Level  3  or  higher. 

18  months  after  contract  award,  the  program  office  conducted  a  CMMI  “Class 
B”  appraisal  on  the  team. 
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Example  Program 

Issues  Identified  i 


PROJECT  MANAGEMENT 

•  Lack  of  project  plans  or  having  only  incomplete, 
conflicting  or  out  of  date  project  plans 

.  Ineffective  use  of  Integrated  Master  Schedule 
as  basis  for  planning/tracking  status  across 
program 

.  Undefined  engineering  and  management 
processes  on  program 

.  Inability  to  track  and  manage  actions  to  closure 

•  Inadequate  cost  estimation  processes,  methods, 
data  and  tools 

•  Inadequate  staffing  and  training  project 
personnel 

•  Tracking  dependencies  between  or  across 
teams  not  defined 

•  Managing  project  data  ad  hoc 

•  Inability  to  proactively  identify  and  manage  risks 


ENGINEERING 

•  Lack  of  understanding  of  the  program’s 
requirements 

•  Inability  to  trace  requirements  to 
architecture/design  or  to  test  plans/procedures 

•  Poor  linkage  of  functional  and  performance 
requirements 

•  Inconsistent  requirements  management  at 
different  levels 

.  No  criteria  for  making  architectural/design 
decisions  among  alternatives 

•  Not  capturing  entire  technical  data  package 
(requirements,  design  and  design  rationale,  test 
results,  etc) 

.  Efficiency  of  design  process/methods  in 
question 

•  Late  definition  of  integration  and  test  procedures 
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Example  Program 

Issues  Identified  2 

SUPPORT 

Difficult  to  identify  items  in  configuration  management  baselines 

Lack  of  ability  to  manage  individual  “versions”  in  incremental  development 

Inability  to  effectively  managing  changes  to  work  products  throughout  lifecycle 

Not  conducting  audits  to  establish/ensure  integrity  of  baselines  throughout  incremental 
engineering  and  development 

Inefficient  change  management  process  (cycle  time,  volume  of  changes) 
Roles/responsibilities  of  change  control  boards  not  defined 
Quality  Assurance  audits  of  products  and  processes  not  consistent 
QA  involvement  in  system  and  software  engineering  processes  not  consistent 
No  metrics  to  manage  engineering  activities  (outside  of  cost/schedule  data) 
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Complexity  in  Modern  Systems 


Many  commercial  products  are  the  result  of  a  complex  mix  of 
subcomponents  engineered  into  a  system 


Most  DoD  weapon  and  information  systems  are  at  least  this 
complex 
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Weapon  System  Complexity 
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System  of  Systems  Complexity 


AWACS 


Collection 
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Acquirer/Supplier  Mismatch 
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Long  History  of  Software  Problems 


“[Acquisition  directives]  do  not  encourage  [iteration].  They  essentially  forbid  it. 
[Standards]  continue  to  reinforce  exactly  the  document-driven  specify-then-build 
approach  that  lies  at  the  heart  of  so  many  DoD  software  problems.” 

“DoD  does  not  have  adequate  career  paths  for  software  professionals.” 

“The  application-knowledgeable,  technical  skilled  leaders  are  the  military’s  limiting 
resource  in  using  today’s  computer  technology.” 

“...the  hardest  part  of  the  software  task  is  the  setting  of  exact  requirements  ... 
including  the  relative  priorities  ...  for  inevitable  tradeoffs.” 

DoD  should  be  aggressively  looking  for  opportunities  to  buy  in  the  civilian  market  tools, 
methods,  environments,  and  application  software.” 

Computer  security  requirements  are  frequently  cited  as  a  reason  why  [COTS] 
software  cannot  be  used.” 


“Today’s  major  problems  with  military  software  development  are  not  technical 
problems,  but  management  problems.” 


“Many  previous  studies  have  provided  an  abundance  of  valid  conclusions  and 
recommendations.  Most  remain  unimplemented.” 

Report  of  the  DSB  Task  Force  on  Military  Software,  1987 
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Systems  and  Software  Engineering 
Organizational  Core  Competencies 


Director,  Systems  & 
Software  Engineering 


CORE  COMPETENCIES 


•  SE  Policy 

•  SE  Guidance 

•  SE  in  Defense  Acquisition 
Guidebook 

•  Technical  Planning 

•  Risk  Management 

•  Reliability  & 
Maintainability 

•  Contracting  for  SE 

•  SoS  SE  Guide 

•  SE  Education  and  Training 

•  DAU  SE  Curriculum 

•  SPRDE  Certification  Rqmt 

•  Corrosion 

•  R-TOC 

•  Value  Engineering 


CORE  COMPETENCIES 


DT&E  Policy 
DT&E  Guidance 

•  T&E  in  Defense 
Acquisition  Guidebook 

•  TEMP  Development 
Process 

DT&E  Education  and 
Training 

•  DAU  DT&E  Curriculum 

•  DT&E  Certification  Rqmt 
Joint  Testing,  Capabilities 
&  Infrastructure 
Targets  Oversight 

Acq  Modeling  &  Simulation 
Energy 

DSOC/Acq  Tech  Task  Force 


CORE  COMPETENCIES 


SWE  and  SA  Policy 
SWE  and  SA  Guidance 

•  SoS,  SA  Guides 

SWE  and  SA  Education  and 
Training 

•  DAU  SW  Acq  Curriculum 

•  Continuous  Learning 
Modules  for  SWE,  SoS,  SA 

Software  Engineering 

•  Acquisition  Support 

•  Software  Engineering 
Institute  (SEI) 

Process  Improvement 

•  CMMI  Sponsor 
DoD/National  Software 
Investment  Strategy 


Est.  Aug  06<^ 


Mark  Schaeffer  SES 

1 

_ i 

v 

1 

1 

_ 

Deputy  Director 
Enterprise  Development 

Deputy  Director 
Developmental  Test 
&  Evaluation 

Deputy  Director 
Software  Engineering  & 
System  Assurance 

Bob  Skalamera  SES 

Chris  DiPetto  SES 

Kristen  Baldwin 

SES 

Deputy  Director 
Assessments  &  Support 


Dave  Castellano 


SES 


CORE  COMPETENCIES 


•  Support  of  ACAT  I  and 
Other  Special  Interest 
Programs  (MDAP,  MAIS) 

•  Assessment  Methodology 
(Program  Support 
Reviews  -  PSRs) 

•  T&E  Oversight  and 
Assessment  of  Operational 
Test  Readiness  (AOTR) 

•  Systems  Engineering  and 
Developmental  Test 
Planning  and  Support 

•  Lean/6-Sigma  Training/Cert 


Acquisition  program  excellence  through  sound  systems  and  software  engineering 
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DoD  Software  Strategy  Summit  to  Focus  Activities 

Software  Acquisition  and  Sustainment  Software  Engineering 


Software  issues  not  addressed  early  in  lifecycle 

Software  requirements  not  well  defined  at  program  start 

Management  has  limited  visibility  into  software  development  processes  and 
status 

Risk  areas  -  single  point  failures  not  adequately  addressed,  e.g.,  single 
software  providers,  incomplete  data  rights,  key  personnel  stability,  life  cycle 
support  of  COTS 

Acquirers  do  not  adequately  address  software  sustainment  and  total  life 
cycle  early  in  the  program 

Some  agencies  contract  before  engineering  is  complete,  prior  to  system 
design  and  development 


Human  Capital 


Weak  linkage  between  software  requirements  and 
capabilities/portfolios 

System  development  methods  do  not  properly  leverage 
software  ability  to  rapidly  field  new  capability 

Systems  and  software  engineering  lifecycles  not  always 
consistent  or  harmonized 

Software  considerations  not  consistently  addressed  in 
architectures 

Inadequate  software  estimating  methods,  e.g.,  COTS/NDI; 
best  practices  not  applied 


Policy 


Experienced  system  &  software  engineers  seem  missing  from 
key  DoD  leadership  positions 

Shortage  of  highly  experienced  software  managers,  architects, 
domain  and  technical  experts 

Eroding  depth  and  breath  of  experience  for  personnel  in  DoD 

Young  people  may  consider  system  and  software  engineering  as 
a  career  dead  end 


PMs  need  assistance  with  software  policy  and  analysis 

Arbitrary  separation  of  weapon  and  information  technology 
software  policies 

Policy  implementation  guidance  and 
follow-up  monitoring  is  limited 

Department  needs  software  group  with  good  expertise  to 
oversee  and  implement  policy 


Emerging  skill  set  may  be  needed  for  future  complex  DoD 
systems,  e.g.,  systems  of  systems 


Need  capability  to  share  policy  and  guidance  information 


October,  2006  I 
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Acquisition  Support  Program 


Vision 

Predictable  success  in  the  acquisition  of  software  and  systems 


Overall  Goal 

A  continuous  program  of  applying  new  software  engineering 
knowledge  and  techniques  to  increasingly  complex  program 
environments  and  amplifying  their  application  through  the  acquisition 
infrastructure  throughout  the  DoD,  Federal  Agency  and  other  acquirer 
communities. 
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Acquisition  Support  Program 

Strategies 


•  Impact  individual  programs  -  work  with  key  DoD,  Federal  Agency, 
and  other  acquisition  programs  to  help  them  meet  their  objectives 


•  Impact  acquisition  organizations  -  help  establish  a  learning 
environment  within  acquisition  organizations 


•  Define,  integrate  and  transfer  knowledge  -  help  improve  the  state  of 
the  practice 


Software  Engineering  Institute 
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ASP  Areas  of  Work 


Mission  Assurance 


Process 


S/W  Engineering 


Systems  Engineering 


Architecture 


Interoperability 


) 


) 


Knowledge 
Integration,  and 
Transfer 


Security 


Improved 

Systems 


Improved  State 
of  Practice 
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ASP  Operational  Plan 


Direct  Benefit  to 
Acquisition  Programs 


Indirect  Benefit  to 
Similar  Programs 


Acquisition 
Support 
Program 

applies 

e  and  Systems 
Technologies 


Feedback  from  direct  support 
and  community  learning 
improves  ASP  practices  &  SEI 
technologies 


Workshops ,  Classes,  Seminars 

Tailored  learning  via 
Acquisition  Communities  of 
Practice 

•  Army,  Navy,  Air  Force,  Defense  and  Intel 
Agencies 

•  Software  Collaborator’s  Network 

•  Conferences 

•  MITRE,  Aerospace 

•  Defense  Acquisition  University 

•  OSD  Best  Practices 

•  Civil  Agencies 

•  Universities 

•  US-UK-AUS  Working  Groups 
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SEI  Acquisition  -  Footprints 


Army 

•  ASSIP,  Future  Combat  Systems,  PEO  Aviation,  AMRDEC  SED,  CECOM  SEC,  AMCOM,  PM 
Aviation,  AMPS/JMPS,  PM  TAPO,  US  Army  Reserve,  PM  FBCB2,  AMRDEC  AADL 

Navy 


•  DDG-1000,  Common  Link  Integrated  Processor,  Littoral  Combat  Ship,  Multi-Mission  Maritime 
Aircraft,  Open  Architecture  and  DASN  IWS 

Air  Force 

•  SAF/AQ,  Standard  Systems  Group,  HRC2SPO,  IDECS,  C-130  AMP,  Joint  Mission  Planning 
System,  MILSATCOM  (AEHF,  FAB-T,  CCS-C,  TSAT),  Space  Radar,  GPS,  SMC  Engineering 
Baseline,  E10A  (MC2A),  ESC  ACE,  Joint  Environmental  Toolkit,  MEECN,  AFRL 

Joint/Other  DoD 

•  Joint  Strike  Fighter,  JSSEO,  MDA,  DFAS 

Intelligence  Agencies 

•  National  Security  Agency,  National  Reconnaissance  Office,  National  Geospatial  Intelligence 
Agency,  Department  of  Homeland  Security 

Civil  Agencies 

•  Internal  Revenue  Service,  Department  of  Veterans  Affairs,  Nuclear  Regulatory  Commission, 
National  Aeronautics  &  Space  Administration,  US  Coast  Guard 
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Themes 


Educating  the  acquirer 

Imparting  requisite  software  knowledge  to  define,  monitor,  and  manage  a  program;  training 
and  mentoring;  effective  teaming. 

Advancing  software-aware  system  engineering 

Advising  on  requirements  engineering  and  management;  system  architecting,  design, 
construction,  and  integration;  verification  and  validation;  and  sustainment  and  refresh 
techniques  that  suit  complex  environments. 

Facilitating  horizontal  integration 

Guiding  the  acquirer  on  development  of  robust  architectures,  interoperable  systems, 
integration  of  disparate  data,  data  mining,  integrating  the  “enterprise,”  etc. 

Overcoming  process  aversion 

Communicating  the  value  of  process,  modeling  processes  to  identify  inefficiencies  and  the 
need  for  improvement. 

Overcoming  technology  aversion 

Understanding  prevalent  attitudes,  ensuring  people  are  considered  in  technology  solutions. 

Tempering  technology  worship 

Performing  robust  risk-benefit  analyses,  defining  feasible  off-ramps. 
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iat'Of' 


SKILLS 

Bridging  the  gap  between  your 
current  crisis  and  software  best 
practices 
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Software  Acquisition  Survival  Skills 


3-day  course  aimed  at  PMs  and  program  office  personnel 
Topics: 

—  Risk  Management 
—  Pre-Award  Activities 
—  Requirements  Management 
—  Systems  Engineering 
—Technical  Evaluation 
—  Software  Architecture 
—  Managing  with  Metrics 
—  Process  Management 
—  Concept  Integration 
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Leveraging  Customer  Engagements 


ASP  uses  customer  engagements  to  improve  the  state  of  the  practice 
of  software  and  systems  acquisition  in  the  following  ways: 


•  Catalyst  -  small  investments  for  large  impact 

•  Integration  -  the  whole  is  greater  than  the  sum  of  the  parts 

•  Packaging  &  Dissemination  -  best  practices  amplified 


_  Software  Engineering  Institute  CamegieMellon 


Acquisition  Support 

©  2007  Carnegie  Mellon  University 


26 


Catalyst:  Acquisition  Strategy  Guidance 


Question  from  ASA/ALT 

•  What  is  the  appropriate  strategy  when  acquiring  software-intensive 
systems? 

Scope 

•  Mine  &  distill  existing  know-how 

•  Provide  workbook,  not  guidebook  -  “it  depends” 

•  Concise,  practical,  repeatable  process  for  DoD  programs 
Funded  as  part  of  Army’s  Strategic  Software  Improvement  Program 
Developed  based  on  DoD  5000;  piloted  with  Army  programs 
Results 

•  Techniques  for  Developing  an  Acquisition  Strategy  by  Profiling  Software 
Risks  (CMU/SEI-2006-TR-002)  and  excel  tool  available  for  download 
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Integration:  Quality  Assessment  of  System  Architectures 
and  their  Requirements  (QUASAR) 

A  method  on  the  use  of  Quality  Cases  for  assessing  the  quality  of 

•  Software-intensive  System/Subsystem  Architectures  and  the 

•  Architecturally-Significant  Requirements  that  drive  them 
Developed  based  on  funded  work  on  the  JSF  program 
Integrated  technology  from 

•  Assurance  Case  research 

•  Quality  Attribute  and  Architecture  Evaluation  research 

•  Method  Development  from  Process  research 

Results 

•  Method  description  documented  for  subsequent  use 
(CMU/SEI-2006-HB-001 ) 

•  Effort  lauded  by  Mike  Bossert,  F-35  Air  System  Architect,  JSF  Program 
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Packaging  &  Dissemination:  Building  Awareness  of  CMMI 
in  the  Acquisition  Community 


NDIA  Workshop  &  Summit  on  CMMI  Use  in 
DoD  Programs 

Piloting  new  and  innovative  ways  to  use 
CMMI  to  encourage  use  of  effective  practices 
on  DoD  programs 

•  AF  programs:  KG  3X,  JET,  JMPS,  TSAT 

•  Navy  programs:  MMA,  CLIP 

•  Intelligence  Community  programs:  NSA, 

NRO 

Results 

•  Understanding  and  Leveraging  a  Supplier’s 
CMMI  Efforts: 

A  Guidebook  for  Acquirers  (CMU/SEI-2007-  -  Mark  Schaeffer 

TR-004)  Director,  Systems  and 

Software  Engineering 


“Level  X 

companies  often  do 
not  perform  at  that 
level  on  all 
programs ” 

“DoD  expects  that  if 
you  have  achieved 
high  maturity,  the 
next  program  will 
perform  at  that 
maturity” 
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Agenda 


State  of  Acquisition 

What  is  the  Department  of  Defense  Doing? 
What  is  the  SEI  Doing? 

Principles  of  Effective  Acquisition 
Summary 


_  Software  Engineering  Institute  CamegieMellon 


Acquisition  Support 

©  2007  Carnegie  Mellon  University 


30 


The  Quest  for  the  “Silver  Bullet” 


Open  Architecture 


Interoperability 


Acquisition  Reform  0  .  ...  ,  ...  ...... 

Total  System  Performance  Responsibility 

Agile  Acquisition 

Evolutionary  Acquisition 

Performance-Based  Acquisition 

Capability-Based  Acquisition 

Net-Centric  Warfare  lnsi9ht  versus  Oversight 

Time-Certain  Development  Service-Based  Acquisition 

Architecture-based  Development 


Systems  Engineering  Revitalization 


Lean  Six  Sigma 
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Principle-Based  Decisions 


“Principle”  Defined: 

The  collectivity  of  moral  or  ethical  standards  or  judgments:  a 
decision  based  on  principle  rather  than  expediency. 


Decisions  to  pursue  a  given  acquisition  approach  should  be  grounded 
on  underlying  principles  designed  to  increase  the  effectiveness  of 
acquiring  and  deploying  systems. 


The  following  describes  the  Seven  Principles  of  Effective  Acquisition. 
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The  Core  Principle:  Open  Communication 


Encouraging  free  flowing  information  at 
and  between  all  stakeholders. 


Enabling  formal,  informal,  and 
impromptu  communication. 


Using  consensus-based  processes  that  value  the  individual 
voice  (bringing  unique  knowledge  and  insight  to  evolving 
mission  capabilities). 
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The  Three  Sustaining  Principles 


Team  Risk  Management 


Continuous  Process  Improvement 


Continuous  Product  Improvement 
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Team  Risk  Management 


Evolving  the  mission  capabilities 
by  continuously  mitigating 
operational,  development, 
and  acquisition  risks. 


All  stakeholders  participating  in  managing  the  project 
by  managing  the  risks. 
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Continuous  Process  Improvement 


Maturing  the  acquisition,  development, 
and  operational  processes  to  meet 
the  mission  objectives. 


Employing  a  common  process  improvement  framework  and 
language  to  align  and  enhance  process  capability. 
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Continuous  Product  Improvement 


Enhancing  the  mission 
through  evolutionary  delivery 
of  enhanced  capabilities. 


Delivering  an  initial  capability  on  the  first  promise  date, 
with  the  demonstrated  capability  to  deliver  improved  or 
updated  capability  in  on  a  regular,  dependable 
schedule. 
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The  Three  Defining  Principles 


Forward-Looking  View 


Global  Perspective 


Shared  Product  Vision 
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Forward-Looking  View 


Seeing  a  common  tomorrow  against 
which  all  stakeholders  can  measure 
potential  breakthroughs  and  risks. 


Managing  project  resources  and  activities  while  anticipating 
uncertainties. 
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Global  Perspective 


Sharing  a  single  mental  model  of  project 
success  that  crosses  all  boundaries 
between  acquirer,  developer, 
and  operator. 


Viewing  enhancements  within  the  — 

context  of  the  operational 
mission. 

Recognizing  both  the  potential  value  of  opportunity  and  the 
potential  impact  of  adverse  effects. 


Shared  Product  Vision 


Developing  and  sustaining  a  common 
conception  of  the  product  being  built  -  one 
that  can  be  stated  simply  and  briefly,  and 
is  founded  on  common  purpose,  shared 
ownership,  and  collective  commitment 
among  the  stakeholders. 


Focusing  on  results. 
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Seven  Principles  of  Effective  Acquisition 
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Agenda 


State  of  Acquisition 

What  is  the  Department  of  Defense  Doing? 
What  is  the  SEI  Doing? 

Principles  of  Effective  Acquisition 
Summary 
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Summary 


The  SEI,  through  the  Acquisition  Support  Program,  works  directly  with 
key  acquisition  programs  to  help  them  meet  their  objectives. 


ASP  looks  for  common  themes  and  solutions  and  packages  them  for 
wider  dissemination  and  use. 
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For  More  Information 


Brian  Gallagher 
Director 

Acquisition  Support  Program 
Telephone:  412-268-7157 
E-mail:  bq@sei.cmu.edu 


Internet: 

http://www.sei.cmu.edu/acquisition-support 


Software  Engineering  Institute 


Carnegie  Mellon 


